Papers by Maximo Eduardo Rulli
Attention Sinks in Diffusion Language Models (2026.findings-acl)
Copied to clipboard
Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, Alessio Devoto
| Challenge: | Masked Diffusion Language Models (DLMs) employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. |
| Approach: | They conduct an empirical analysis of DLM attention patterns focusing on the attention sinking phenomenon . they find that DLMs also exhibit attention sinks, but with distinct characteristics . |
| Outcome: | The proposed models employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. |